PMC

(A) Experiment 1. Participants were scanned during three 7-minute runs during which the audiovisual (AV), visual-only (V) and auditory-only (A) stimuli were presented. In each run participants were shown 27 stimuli (3 emotions: sad, happy, and surprise X 3 displays X 3 modalities: audiovisual, auditory, and visual) for a total of 81 presentations. The audiovisual condition consisted of the original musical displays (2 s) in which auditory and visual signals were in synchrony, as illustrated in the temporal profiles at the bottom of the figure. The visual-only condition showed the visual displays extracted from the original audiovisual displays without any sound. The auditory displays were presented by showing a black screen together with the auditory signal. (B) Experiment 2. Participants were scanned during three 6-minute runs during which the emotionally matching (M) and mismatching (MM) displays were presented. In each run participants were shown 24 stimuli, of which 6 were emotionally matching and 18 were emotionally mismatching, for a total of 72 presentations. The emotionally matching displays combined visual and auditory signals that expressed the same emotion but were extracted from different original audiovisual displays. In contrast, the emotionally mismatching displays combined visual and auditory information that expressed different emotions and was extracted from different original audiovisual displays. In this way both types of emotionally matching and mismatching stimuli were presented as asynchronous visual and auditory signals, as illustrated in the temporal profiles at the bottom of the figure.

Fig 6A displays the results of Repeated measures analysis in heart rates when the auditory stimuli (music) were positive. Fig 6B displays the results of Repeated measures analysis in heart rates when the auditory stimuli (music) were negative. Fig 6C displays the results of Repeated measures analysis in skin temperatures when the auditory stimuli (music) were positive. Fig 6D displays the results of Repeated measures analysis in skin temperatures when the auditory stimuli (music) were negative. Fig 6E displays the results of Repeated measures analysis in EMG root mean squares when the auditory stimuli (music) were positive. Fig 6F displays the results of Repeated measures analysis in EMG root mean squares when the auditory stimuli (music) were negative.

Schematic explanation of creating bimodal incongruent stimuli. Visual angry displays were combined with auditory happy displays, while visual happy were combined with auditory angry displays. Two types of auditory filtering were used in Experiment 1 (see Section 3.3), but only low-pass filtering was used in Experiment 2. For illustrative purposes, red represents happy displays and red-happy displays.

All Experiments tested unimodal (auditory only, first column, or visual only, second column), multisensory (congruent auditory-visual, third column) and incongruent or intersensory auditory-visual conditions (auditory intersensory, fourth column and visual intersensory fifth column). In the control conditions, the target (4^th stimulus in a stream of five stimuli) differed from the standards (stimulus 1, 2, 3 and 5; all 500 ms) in duration only. In the test conditions, the target differed from the standards in both feature and duration. In Experiment 1 (‘Loom’, first row) and Experiment 2 (‘Recede’, second row), the same control conditions were used, where standards were 500 ms discs or pure tones in visual and auditory displays, respectively. In the Loom tests, auditory standards were 500 ms pure tones, and auditory targets were upward going FM sweeps of varying duration; visual standards were 500 ms discs and visual targets were looming discs of different duration; auditory and visual conditions were combined in the multisensory condition. In the Recede tests, the target was a downward going FM sweep or a receding disc in the auditory and visual sensory modalities, respectively. In the control of Experiment 3 (‘Reverse’, third row), the auditory standards were upward FM sweeps and the visual standards were 500 ms looming discs. In the Reverse tests, the oddballs were a steady disc and a pure tone of variable duration in visual and auditory displays, respectively. The Loom, Recede and Reverse intersensory conditions consisted in presenting congruent auditory-visual standards but incongruent auditory-visual targets. An oddball was introduced in the sensory modality which was to be ignored. In the auditory intersensory conditions, participants evaluated the auditory target while neglecting visual inputs; conversely, in the visual intersensory conditions, participants evaluated the visual target while ignoring the auditory inputs. In the Loom auditory (first row, fourth column) and visual intersensory (first row, fifth column) conditions, the oddball was a looming disc and an upward FM sweep, respectively. In the Recede auditory (second row, fourth column) and visual intersensory (second row, fifth column) conditions, the oddball was a receding or a downward FM sweep, respectively. In the Reverse auditory (third row, fourth column) and visual intersensory (third row, fifth column) conditions, the oddball was a steady disc or a tone, respectively.